Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Ingest server #370

Draft
wants to merge 37 commits into
base: main
Choose a base branch
from
Draft

feat: Ingest server #370

wants to merge 37 commits into from

Conversation

stevekeay
Copy link
Contributor

@stevekeay stevekeay commented Oct 9, 2024

WIP because still need to add yaml and figure out password handling.

The device that gets created in Nautobot looks like the blelow - it has all the info we are accustomed to, except for "U" height and CORE number. The hostname format also looks different.

image

image

Sample logs from running this thing:

2024-10-09 22:20:10,198 - __main__ - INFO - /home/steve.keay/p/understack/python/understack-workflows/understack_workflows/main/ingest_or_refresh_baremetal_node.py starting for bmc_ip_address='10.46.96.156'
2024-10-09 22:20:10,200 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): nautobot.dev.undercloud.rackspace.net:443
2024-10-09 22:20:10,512 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/ HTTP/1.1" 200 839
2024-10-09 22:20:10,520 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): 10.46.96.156:443
2024-10-09 22:20:11,648 - urllib3.connectionpool - DEBUG - https://10.46.96.156:443 "GET /redfish/v1/Systems/System.Embedded.1/ HTTP/1.1" 200 2757
2024-10-09 22:20:11,659 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): 10.46.96.156:443
2024-10-09 22:20:12,364 - urllib3.connectionpool - DEBUG - https://10.46.96.156:443 "GET /redfish/v1/Managers/iDRAC.Embedded.1/EthernetInterfaces/NIC.1 HTTP/1.1" 200 625
2024-10-09 22:20:12,370 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): 10.46.96.156:443
2024-10-09 22:20:13,081 - urllib3.connectionpool - DEBUG - https://10.46.96.156:443 "GET /redfish/v1/Systems/System.Embedded.1/EthernetInterfaces/ HTTP/1.1" 200 253
2024-10-09 22:20:13,089 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): 10.46.96.156:443
2024-10-09 22:20:13,695 - urllib3.connectionpool - DEBUG - https://10.46.96.156:443 "GET /redfish/v1/Systems/System.Embedded.1/EthernetInterfaces/NIC.Integrated.1-1 HTTP/1.1" 200 475
2024-10-09 22:20:13,702 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): 10.46.96.156:443
2024-10-09 22:20:14,310 - urllib3.connectionpool - DEBUG - https://10.46.96.156:443 "GET /redfish/v1/Systems/System.Embedded.1/EthernetInterfaces/NIC.Integrated.1-2 HTTP/1.1" 200 475
2024-10-09 22:20:14,316 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): 10.46.96.156:443
2024-10-09 22:20:14,924 - urllib3.connectionpool - DEBUG - https://10.46.96.156:443 "GET /redfish/v1/Systems/System.Embedded.1/EthernetInterfaces/NIC.Slot.1-1 HTTP/1.1" 200 477
2024-10-09 22:20:14,930 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): 10.46.96.156:443
2024-10-09 22:20:15,420 - urllib3.connectionpool - DEBUG - https://10.46.96.156:443 "GET /redfish/v1/Systems/System.Embedded.1/EthernetInterfaces/NIC.Slot.1-2 HTTP/1.1" 200 477
2024-10-09 22:20:15,429 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): 10.46.96.156:443
2024-10-09 22:20:16,049 - urllib3.connectionpool - DEBUG - https://10.46.96.156:443 "GET /redfish/v1/Systems/System.Embedded.1/NetworkPorts/Oem/Dell/DellSwitchConnections/ HTTP/1.1" 200 549
2024-10-09 22:20:16,054 - __main__ - INFO - Discovered ChassisInfo(manufacturer='Dell Inc.', model_number='PowerEdge R7615', serial_number='33GSW04', bmc_ip_address='10.46.96.156', bios_version='1.6.10', interfaces=[InterfaceInfo(name='iDRAC', description='Dedicated iDRAC interface', mac_address='A8:3C:A5:35:43:86', ipv4_address=IPv4Interface('10.46.96.156/26'), remote_switch_mac_address='C4:4D:84:48:61:80', remote_switch_port_name='GigabitEthernet1/0/3'), InterfaceInfo(name='NIC.Integrated.1-1', description='Integrated NIC 1 Port 1', mac_address='D4:04:E6:4F:8D:B4', ipv4_address=None, remote_switch_mac_address='C4:7E:E0:E4:10:7F', remote_switch_port_name='Ethernet1/5'), InterfaceInfo(name='NIC.Integrated.1-2', description='Integrated NIC 1 Port 2', mac_address='D4:04:E6:4F:8D:B5', ipv4_address=None, remote_switch_mac_address='C4:7E:E0:E4:32:DF', remote_switch_port_name='Ethernet1/5'), InterfaceInfo(name='NIC.Slot.1-1', description='NIC in Slot 1 Port 1', mac_address='14:23:F3:F5:25:F0', ipv4_address=None, remote_switch_mac_address='C4:7E:E0:E4:32:DF', remote_switch_port_name='Ethernet1/6'), InterfaceInfo(name='NIC.Slot.1-2', description='NIC in Slot 1 Port 2', mac_address='14:23:F3:F5:25:F1', ipv4_address=None, remote_switch_mac_address='C4:7E:E0:E4:10:7F', remote_switch_port_name='Ethernet1/6')])
2024-10-09 22:20:16,459 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "POST /api/graphql/ HTTP/1.1" 200 23
2024-10-09 22:20:16,461 - understack_workflows.nautobot_device - INFO - Device 33GSW04 not in Nautobot, creating
2024-10-09 22:20:17,478 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "POST /api/graphql/ HTTP/1.1" 200 243
2024-10-09 22:20:17,985 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "POST /api/graphql/ HTTP/1.1" 200 244
2024-10-09 22:20:18,917 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "POST /api/graphql/ HTTP/1.1" 200 243
2024-10-09 22:20:18,919 - understack_workflows.nautobot_device - INFO - Server device: {'status': {'name': 'Staged'}, 'role': {'name': 'server'}, 'device_type': {'manufacturer': {'name': 'Dell'}, 'model': 'PowerEdge R7615'}, 'name': 'Dell-33GSW04', 'serial': '33GSW04', 'rack': '1ccd4b4a-7ba3-4557-b1ad-1ba87aee96a6', 'location': 'da47f07f-b66a-4f0c-b780-4be8498e6129'}
2024-10-09 22:20:19,634 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "POST /api/dcim/devices/ HTTP/1.1" 201 2886
2024-10-09 22:20:20,241 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "POST /api/graphql/ HTTP/1.1" 200 435
2024-10-09 22:20:20,761 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/dcim/interfaces/?device=52219630-9803-4827-a139-2d8e39b42361&name=iDRAC HTTP/1.1" 200 1472
2024-10-09 22:20:20,763 - understack_workflows.nautobot_device - INFO - Updating existing interface iDRAC 40e5e8f0-b498-404c-8512-28a9b6363e22 in Nautobot
2024-10-09 22:20:21,818 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "PATCH /api/dcim/interfaces/40e5e8f0-b498-404c-8512-28a9b6363e22/ HTTP/1.1" 200 1501
2024-10-09 22:20:22,035 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/ipam/ip-addresses/?address=10.46.96.156 HTTP/1.1" 200 1173
2024-10-09 22:20:22,035 - understack_workflows.nautobot_device - INFO - Nautobot IP 312d73f7-5b19-4cb4-b098-05d913ccef2d already exists for 10.46.96.156/26
2024-10-09 22:20:22,254 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/ipam/ip-address-to-interface/?ip_address=312d73f7-5b19-4cb4-b098-05d913ccef2d&interface=40e5e8f0-b498-404c-8512-28a9b6363e22 HTTP/1.1" 200 52
2024-10-09 22:20:22,453 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "POST /api/ipam/ip-address-to-interface/ HTTP/1.1" 201 916
2024-10-09 22:20:22,455 - understack_workflows.nautobot_device - INFO - Associated IP address 312d73f7-5b19-4cb4-b098-05d913ccef2d with iDRAC
2024-10-09 22:20:23,191 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/dcim/interfaces/?device=912d38b1-1194-444c-8e19-5f455e16082e&name=GigabitEthernet1%2F0%2F3 HTTP/1.1" 200 1721
2024-10-09 22:20:23,194 - understack_workflows.nautobot_device - INFO - Interface iDRAC connects to f20-2-1d.iad3.rackspace.net GigabitEthernet1/0/3
2024-10-09 22:20:23,394 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/dcim/cables/?termination_a_id=4d010e0f-3135-4769-8bb0-71ba905edf01&termination_b_id=40e5e8f0-b498-404c-8512-28a9b6363e22 HTTP/1.1" 200 52
2024-10-09 22:20:24,652 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "POST /api/dcim/cables/ HTTP/1.1" 201 1429
2024-10-09 22:20:24,654 - understack_workflows.nautobot_device - INFO - Created cable 6492d0dd-12da-454e-a85e-b9f34d1d7fe3 in Nautobot
2024-10-09 22:20:24,959 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/dcim/interfaces/?device=52219630-9803-4827-a139-2d8e39b42361&name=NIC.Integrated.1-1 HTTP/1.1" 200 52
2024-10-09 22:20:25,676 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "POST /api/dcim/interfaces/ HTTP/1.1" 201 1539
2024-10-09 22:20:25,679 - understack_workflows.nautobot_device - INFO - Created interface NIC.Integrated.1-1 26c1bd1a-ca2d-4eda-bc63-9c1655c73c38 in Nautobot
2024-10-09 22:20:26,154 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/dcim/interfaces/?device=275ef491-2b27-4d1b-bd45-330bd6b7e0cf&name=Ethernet1%2F5 HTTP/1.1" 200 1694
2024-10-09 22:20:26,157 - understack_workflows.nautobot_device - INFO - Interface NIC.Integrated.1-1 connects to f20-2-1.iad3.rackspace.net Ethernet1/5
2024-10-09 22:20:26,597 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/dcim/cables/?termination_a_id=f9a5cc87-d10a-4827-99e8-48961fd1d773&termination_b_id=26c1bd1a-ca2d-4eda-bc63-9c1655c73c38 HTTP/1.1" 200 52
2024-10-09 22:20:27,199 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "POST /api/dcim/cables/ HTTP/1.1" 201 1429
2024-10-09 22:20:27,201 - understack_workflows.nautobot_device - INFO - Created cable 45f305ea-9694-479d-88a5-0fcda3dca51b in Nautobot
2024-10-09 22:20:27,826 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/dcim/interfaces/?device=52219630-9803-4827-a139-2d8e39b42361&name=NIC.Integrated.1-2 HTTP/1.1" 200 52
2024-10-09 22:20:28,302 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "POST /api/dcim/interfaces/ HTTP/1.1" 201 1539
2024-10-09 22:20:28,304 - understack_workflows.nautobot_device - INFO - Created interface NIC.Integrated.1-2 38881b35-0567-4da5-b32e-b6c0c9ab26e1 in Nautobot
2024-10-09 22:20:29,260 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/dcim/interfaces/?device=05f6715a-4dbe-4fd6-af20-1e73adb285c2&name=Ethernet1%2F5 HTTP/1.1" 200 1694
2024-10-09 22:20:29,262 - understack_workflows.nautobot_device - INFO - Interface NIC.Integrated.1-2 connects to f20-2-2.iad3.rackspace.net Ethernet1/5
2024-10-09 22:20:29,436 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/dcim/cables/?termination_a_id=2148cf50-f70e-42c9-9f68-8ce98d61498c&termination_b_id=38881b35-0567-4da5-b32e-b6c0c9ab26e1 HTTP/1.1" 200 52
2024-10-09 22:20:30,899 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "POST /api/dcim/cables/ HTTP/1.1" 201 1429
2024-10-09 22:20:30,902 - understack_workflows.nautobot_device - INFO - Created cable c3bc7ca4-0589-4e73-82c4-6843c624d9bf in Nautobot
2024-10-09 22:20:31,205 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/dcim/interfaces/?device=52219630-9803-4827-a139-2d8e39b42361&name=NIC.Slot.1-1 HTTP/1.1" 200 52
2024-10-09 22:20:32,946 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "POST /api/dcim/interfaces/ HTTP/1.1" 201 1518
2024-10-09 22:20:32,949 - understack_workflows.nautobot_device - INFO - Created interface NIC.Slot.1-1 65c26b0e-4fd2-43e8-ad45-dfd4fff5118a in Nautobot
2024-10-09 22:20:33,970 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/dcim/interfaces/?device=05f6715a-4dbe-4fd6-af20-1e73adb285c2&name=Ethernet1%2F6 HTTP/1.1" 200 1694
2024-10-09 22:20:33,972 - understack_workflows.nautobot_device - INFO - Interface NIC.Slot.1-1 connects to f20-2-2.iad3.rackspace.net Ethernet1/6
2024-10-09 22:20:34,149 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/dcim/cables/?termination_a_id=f72bb830-3f3c-4aba-b7d5-9680ea4d358e&termination_b_id=65c26b0e-4fd2-43e8-ad45-dfd4fff5118a HTTP/1.1" 200 52
2024-10-09 22:20:35,213 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "POST /api/dcim/cables/ HTTP/1.1" 201 1429
2024-10-09 22:20:35,216 - understack_workflows.nautobot_device - INFO - Created cable 5552210b-8ab3-4a34-ab45-25e5f2651569 in Nautobot
2024-10-09 22:20:35,478 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/dcim/interfaces/?device=52219630-9803-4827-a139-2d8e39b42361&name=NIC.Slot.1-2 HTTP/1.1" 200 52
2024-10-09 22:20:35,917 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "POST /api/dcim/interfaces/ HTTP/1.1" 201 1518
2024-10-09 22:20:35,919 - understack_workflows.nautobot_device - INFO - Created interface NIC.Slot.1-2 96a197a0-9aa9-4691-90b8-7f38f4c3431b in Nautobot
2024-10-09 22:20:36,428 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/dcim/interfaces/?device=275ef491-2b27-4d1b-bd45-330bd6b7e0cf&name=Ethernet1%2F6 HTTP/1.1" 200 1694
2024-10-09 22:20:36,431 - understack_workflows.nautobot_device - INFO - Interface NIC.Slot.1-2 connects to f20-2-1.iad3.rackspace.net Ethernet1/6
2024-10-09 22:20:36,602 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "GET /api/dcim/cables/?termination_a_id=c210be75-1038-4ba3-9923-60050e1c5362&termination_b_id=96a197a0-9aa9-4691-90b8-7f38f4c3431b HTTP/1.1" 200 52
2024-10-09 22:20:37,181 - urllib3.connectionpool - DEBUG - https://nautobot.dev.undercloud.rackspace.net:443 "POST /api/dcim/cables/ HTTP/1.1" 201 1429
2024-10-09 22:20:37,183 - understack_workflows.nautobot_device - INFO - Created cable b026db96-8b34-4525-8d14-571d7abe9144 in Nautobot
2024-10-09 22:20:37,183 - __main__ - INFO - /home/steve.keay/p/understack/python/understack-workflows/understack_workflows/main/ingest_or_refresh_baremetal_node.py complete successfully for 10.46.96.156

Steve Keay added 5 commits October 9, 2024 21:51
This duplicates some existing code which will be cleaned up later

The intention is to consolidate multiple steps in a single python script, to
simplify information sharing.
@stevekeay stevekeay marked this pull request as draft October 9, 2024 21:26
Copy link

github-actions bot commented Oct 9, 2024

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines Covered Coverage Threshold Status
178 85 48% 0% 🟢

New Files

No new covered files...

Modified Files

No covered modified files...

updated for commit: d577b00 by action🐍

from understack_workflows.bmc import Bmc
from understack_workflows.helpers import setup_logger

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't be here.

Steve Keay added 5 commits October 21, 2024 10:28
ideally this would fetch the actual hardware type but that requires an
additional api call and maybe this will all get replaced with sushy calls anyhow
so I am keeping it simple for now.
Steve Keay added 4 commits October 22, 2024 16:12
We are using nexus switches whose incorrect LLDP implementations give us the
interface mac instead of the chassis mac.  Therefore, accept either from the
switch.  Assume that nautobot always contains the "base" mac address of the
switch.

When finding the switch in nautobot that corresponds to the MAC address that was
reported to us via LLDP, first try the address verbatim, and if that is not
present in nautobot, calculate the "base" mac address (subtracting the port
number from the MAC address reported by LLDP) and try that instead.
This was failing on certain data but we are not using the returned data anywhere
so we can just remove it from the query.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants